Skip to content

docs: audit the documentation set against the shipped 2.1 engine - #447

Merged
MPCoreDeveloper merged 6 commits into
masterfrom
perf/autonomous-20261011
Oct 11, 2026
Merged

MPCoreDeveloper merged 6 commits into
masterfrom
perf/autonomous-20261011

Conversation

@MPCoreDeveloper

Copy link
Copy Markdown
Owner

docs: audit the documentation set against the shipped 2.1 engine and pin the defect classes with tests

Sweep: every backticked identifier, file path and package id in docs/** diffed against the
src/** symbol sets, the project list and Test-Path, then each suspect read at its call site.

Corrected (each verified in source, never inferred):

  • docs/server/CONFIGURATION_SCHEMA.md rewritten: the server registers JSON configuration
    sources only (no TOML/YAML parser exists in the repository), Databases is an array, and the
    real Logging/Performance/Security/ConnectionPool/TenantQuotas/Projections surface is
    documented with the Program.cs startup validation (TLS cannot be disabled, 32-char JWT
    secret, private key required unless the certificate is a .pfx).
  • docs/internals/OPTIMIZER_GUIDE.md: removed SubqueryOptimizer, JoinReorderer, OptimizerCache,
    OptimizerIntegration, QueryOptimizerExtensions, InvalidateOptimizationCache, FilterExecutor,
    PredicateAnalyzer and the unmeasured "Execution Impact" table; documents the real
    Optimization.QueryOptimizer + CostEstimator, the Database/Caching plan cache, the subquery
    classifier/planner/executor/cache, and the honest placeholders (JoinsReordered = hasJoins,
    SubqueriesEliminated = !hasSubquery, hash join disabled so joins are nested-loop).
  • docs/QUERY_PLAN_CACHE.md: GetPlanCacheStats()/ClearPlanCache() are internal
    (InternalsVisibleTo for tests/benchmarks only) - no longer shown as consumer code.
  • docs/manual/{migration,providers}.md: the FluentMigrator integration ships in
    SharpCoreDB.Extensions (AddSharpCoreDBFluentMigrator); there is no SharpCoreDB.FluentMigrator
    package.
  • docs/Vectors/TECHNICAL_SPEC.md: SharpCoreDB.EntityFrameworkCore has no vector support at all
    and no project references Microsoft.Extensions.AI.
  • docs/sharpcoredb-known-issues.md: verified item by item against 2.1 (items 2-5 fixed in the
    engine, 6 stays the default with the UseSqliteIntegerAffinity opt-in); the non-existent
    DurabilityMode.FlushToDisk is corrected; every item annotated.
  • Dead references removed: INDEX.md (4), serialization/README.md (2),
    graphrag/EF_CORE_DOCUMENTATION_INDEX.md (3), analytics/README.md, the issue-56 status table,
    Option.FromNullOrEmpty -> Prelude.Optional, stale v1.7.0 titles, and the never-created
    PERFORMANCE_VALIDATION_AND_ADVICE_CSHARP15_NET11.md pointer.

Guards (DocumentationConsistencyTests, red->green by planting each defect): index paths
resolve; no TOML sample or "Supported Formats: TOML" claim in the server guide; internal
plan-cache APIs not presented as public; every known-issue item annotated; every
install-snippet package id resolves to a project or central package version.

Docs suite 9/0; core suite 2004/0 failed/0 skipped in 89,2 s; build 0 errors.

…ects they exposed

Restamp and re-tabulate the six published pages (README, docs/BENCHMARK_RESULTS.md, docs/manual/performance.md §7.1, docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md, docs/benchmarks/default-config-pk.md and the plan's row 10) from one same-run set at 9279408 / net11.0: default posture fair-PK INSERT 1,17x (1,03-1,27) / READ 1,13x (1,12-1,15) / UPDATE 0,53x (0,42-0,53) / DELETE 0,47x (0,46-0,62) against that run's own SQLite, plus the record-layout A/B (fixed width 1,2-1,5x ahead of legacy variable length on UPDATE/DELETE at a 1,2-1,9x per-record-encryption cost) and the document-CRUD ladder. Every published ratio is the same-run one, because SQLite's reference drifts between sessions and single-run absolutes are not comparable across them.

The correction the publication carries: the "fair-PK layout ahead of SQLite on UPDATE (0,8x) / DELETE (0,5x)" claims came from the window in which the comparator left the SQLite arm allocating a command and two parameters per row. With one prepared command SQLite's own PK references are 3,05x higher (288.108 -> 878.557 UPDATE, 385.116 -> 1.172.704 DELETE), so those two cells are behind, the earlier reference band is superseded rather than better, and the pages now state the direction the measurement gives (improved from 0,39x / 0,31-0,35x). Two further published figures are corrected with their cause: the inline-capacity default is 24 bytes, not the 16 the README still said (+34 % on the batched multi-row INSERT shape, fair-PK INSERT ratio 0,63x -> 0,87x), and the "top v2.1 priority" line is now the historical sentence it was plus where the ratio stands.

Three harness defects, found while publishing and fixed here. BuildConfig's layout switch was `AutoFixedWidthRecords = !fixedWidth`, which cannot express "legacy variable length" because the product default is true, so an arm that merely declined to force the layout still got it for any new PRIMARY KEY table: both layout arms measured ~3 % apart (223.728/562.562 against 230.591/582.384) while the banner promised the layout the plan last measured 2,3x away on UPDATE. It is now an explicit `legacyVariableLength` parameter scoped to that one call site and defaulting to false. The --pk [diag] line printed the requested parameters instead of the resolved config, so the encrypted stock arm read `noEncrypt=True` ("no encryption") on the arm whose whole point is per-record encryption at rest; it now prints config.NoEncryptMode / FixedWidthRecordLayout / AutoFixedWidthRecords (the stock arm reads `noEncrypt=False autoFixedWidth=True atRest=True inline=24`). RunPkMedian builds its own median object and never stamped Reps, so every pk_comparative_*.json archive - the archive the plan publishes beside the fair-PK table so a reader can reproduce the protocol - described a 3-rep median after 8 discarded warm-ups as `"Reps": 1`.

Both are guarded red->green by BenchmarkProtocolArchiveTests (Category=Manual, skips when the harness exe is absent): PkLegacyArm_ResolvesToVariableLengthRecords_AsItsBannerClaims and PkComparisonArchive_RecordsTheMeasuredRepCount_NotTheSingleShotDefault. Validation: core suite 1998 / 0 failed / 0 errors / 0 skipped through the MTP exe with CI's filter, test project builds with 0 errors, 2 / 0 on the guards. No engine code, config default, threshold, package or version metadata changed (legacyVariableLength defaults false, so no existing mode's DatabaseConfig moves), no baseline re-recorded, nothing weakened to publish a number. The four archives the pages cite are committed beside them; the rest of the untracked results stay untracked. Local commit on a feature branch, not pushed.
…t-line docs with it

The README was a 2.1-line page in its headers and a 2.0-line page in its instructions: the
Quick Start installed 2.0.0.3 -- the parked net10 line -- in all three of its snippets, so a
.NET 11 user following it got the wrong line entirely. Fixed, and the rest of the page brought
onto the line it describes.

README
- Quick Start (core, Server.Core, Client) and the package block install 2.1.0.
- What is new in v2.x -> What is new in v2.1.0: split into storage/record layout,
  SQLite-parity SQL semantics, performance/allocation, server and toolchain, with the v2.0-era
  work kept and labelled Carried from the v2.0.x line. Every claim was checked against the
  commits reachable from the v2.1.0 tag -- the 24-byte inline capacity (2026-09-23) and the
  1 MiB .scdb growth default (2026-09-24) predate the publish and ship in it; the SELECT
  version() fix landed after the tag and is therefore not claimed as shipped.
- The SIMD bullet claimed Vector<T> batch filtering, which this line does not use:
  Query/SimdFilter.cs is intrinsics-only (AVX-512 -> AVX2 -> SSE2 -> scalar, int32/double, the
  512-bit tier gated at 1.024 elements) as .github/SIMD_STANDARDS.md requires.
- Core-suite count 1,995 (promotion record) -> 1,998 / 0 failed, re-measured on this line.
- New Requirements and the two release lines table: TFM, language, SDK pin, pipeline, package
  version and status per line, plus SQLite parity, the SIMD standard and the durability rails.
- The package block lists the complete packable set of 27 (was 19) and no longer warns that a
  stable 2.1.0 needs the prerelease filter.
- Previous releases gains 2.0.0.3.

The same defect (a current-line document pinning a superseded install), fixed where the sweep
found it
- docs/manual/quickstart.md, docs/manual/README.md, docs/scdb/PRODUCTION_GUIDE.md,
  docs/analytics/README.md, docs/cqrs/README.md, docs/graphrag/README.md: install pins
  2.0.0 -> 2.1.0 (PRODUCTION_GUIDE also: should show 10.0 or higher -> 11.0).
- docs/INDEX.md: v2.0 labels -> v2.1 (Performance Guide, the manual, the changelog reference
  [2.0.0.2] -> [2.1.0]).
- docs/BENCHMARK_RESULTS.md: the executive summary called the engine a .NET 10 database.

CHANGELOG
- The [Unreleased] bucket now states where it ends: everything it collected up to the v2.1.0
  tag (2026-10-10) shipped in [2.1.0]; only the entries dated 2026-10-11 are unreleased.
- [2.1.0] engine behaviour is unchanged from 2.1.0-RC.3 is qualified to the promotion commit
  itself -- the engine work between the RC.3 tag and the v2.1.0 tag had already landed and is
  part of these packages.

Validation: build 0 errors (287 warnings, the standing set); DocumentationConsistencyTests 3/3;
core suite 1998 / 0 failed / 0 errors / 0 skipped on the CI filter in 90,1 s -- the count the
README now publishes. No code, config, threshold or version moved.
…AOT at or behind the JIT on 12 of 16 cells

Session 78 measured exactly this on 2026-10-09 and closed plan item S4 REJECTED; three maintained
documents still sold the un-measured expectation and the file users read for benchmark status had no
AOT arm at all.

The measurement (reported, not re-run): the comparative harness published with -p:PublishAot=true
(native win-x64 image, 14.023.680 B) and run as that binary against its own JIT build -- fresh
processes, identical options, same box, all 16 cells. Per-cell AOT / JIT, median of the paired runs,
in INSERT / READ / UPDATE / DELETE order: legacy plaintext 1,00 / 0,95 / 0,88 / 1,25; fixed-width
plaintext 0,96 / 0,98 / 0,93 / 1,08; fixed-width at rest 0,99 / 0,90 / 0,89 / 0,92; SQLite control on
the same rig 1,01 / 0,98 / 0,99 / 0,99. 12 of 16 cells strictly below 1,00, one at exactly 1,00; the
control is what makes the 2-12 % READ/UPDATE deficits readable, and the only clear AOT lead (legacy
DELETE 1,25) sits at n=2 processes with a 0,92-1,38 pair spread. So NativeAOT is a deployment option
here (image size, startup, no JIT), not a speed lever.

Fixed -- the same refuted expectation in four places:
- README: the Native AOT readiness bullet now states what AOT is worth, not only that it works.
- docs/manual/performance.md 7.10: the row listed "AOT interface-dispatch improvements -> faster
  interface-heavy storage paths under NativeAOT" among the v2.1 items; it now reads measured, no win.
- docs/performance/V2_PERFORMANCE_PLAN.md 4.1: keeps the vendor expectation, marks it measured and
  not transferring on our workload.
- docs/BENCHMARK_RESULTS.md: gains a NativeAOT arm section -- per-cell table, reproducible publish
  command, the two harness defects that nearly published the opposite verdict, and pointers to plan
  S4, worklog session 78 and the archived runs committed in 5dda8a2.

docs/CHANGELOG.md [Unreleased] -> Fixed records the correction with the numbers.

Validation: build 0 errors; DocumentationConsistencyTests 3/3; core suite 1998 / 0 failed / 0 errors /
0 skipped on the CI filter. Docs only -- no code, config, threshold, test or version moved.
… has no dangling links

The production guide documented an engine that was never shipped: CHECKPOINT SQL, Database.BackupAsync, GetStatistics()/GetPerformanceMetrics(), EnableWalCheckpointing, WalCheckpointIntervalMs, WalDurabilityMode.Fsync, sidecar-*.wal incremental backups, and db.CreateUser/db.Login (which single-file mode rejects with NotSupportedException). Config samples passed DatabaseOptions where the Database constructor takes DatabaseConfig, and CorruptionDetector/RepairTool were handed a DI service provider instead of the SingleFileStorageProvider they require.

Every option, method and type in docs/scdb/ was read out of src/; the guide is now split by type (DatabaseOptions vs DatabaseConfig), the WAL is described as a region inside the .scdb file (db.Flush(), no CHECKPOINT statement), monitoring uses GetStorageStatistics/GetQueryCacheStatistics/GetPageCacheStatistics/GetDatabaseStatistics, and the two unsourced ratio claims (7,765x, 1.37x) are gone in favour of pointers to the measured pages. docs/scdb/README.md's C# 15 samples no longer illustrate the unused ScdbFile prototype.

32 relative links across 11 files pointed at files that do not exist (SCDB_FILE_FORMAT_DESIGN.md, DATABASE_ARCHITECTURE.md, GraphTraversalPerformanceBenchmarks.cs, EFCORE_COLLATE_COMPLETE.md and others); every one now resolves, or names tests/benchmarks/ in text where no artefact is checked in.

Red->green: DocumentationConsistencyTests.DocumentationMarkdownLinks_ShouldResolveToExistingFiles reported 32 dangling links before the sweep and 0 after, and its sensitivity was demonstrated by planting one link (__link_probe.md:3) and watching it fail on exactly that line. Docs suite 4/4; core suite 1999 / 0 failed / 0 errors / 0 skipped. Local commit on perf/autonomous-20261011, never pushed.
…pin the defect classes with tests

Sweep: every backticked identifier, file path and package id in docs/** diffed against the
src/** symbol sets, the project list and Test-Path, then each suspect read at its call site.

Corrected (each verified in source, never inferred):
- docs/server/CONFIGURATION_SCHEMA.md rewritten: the server registers JSON configuration
  sources only (no TOML/YAML parser exists in the repository), Databases is an array, and the
  real Logging/Performance/Security/ConnectionPool/TenantQuotas/Projections surface is
  documented with the Program.cs startup validation (TLS cannot be disabled, 32-char JWT
  secret, private key required unless the certificate is a .pfx).
- docs/internals/OPTIMIZER_GUIDE.md: removed SubqueryOptimizer, JoinReorderer, OptimizerCache,
  OptimizerIntegration, QueryOptimizerExtensions, InvalidateOptimizationCache, FilterExecutor,
  PredicateAnalyzer and the unmeasured "Execution Impact" table; documents the real
  Optimization.QueryOptimizer + CostEstimator, the Database/Caching plan cache, the subquery
  classifier/planner/executor/cache, and the honest placeholders (JoinsReordered = hasJoins,
  SubqueriesEliminated = !hasSubquery, hash join disabled so joins are nested-loop).
- docs/QUERY_PLAN_CACHE.md: GetPlanCacheStats()/ClearPlanCache() are internal
  (InternalsVisibleTo for tests/benchmarks only) - no longer shown as consumer code.
- docs/manual/{migration,providers}.md: the FluentMigrator integration ships in
  SharpCoreDB.Extensions (AddSharpCoreDBFluentMigrator); there is no SharpCoreDB.FluentMigrator
  package.
- docs/Vectors/TECHNICAL_SPEC.md: SharpCoreDB.EntityFrameworkCore has no vector support at all
  and no project references Microsoft.Extensions.AI.
- docs/sharpcoredb-known-issues.md: verified item by item against 2.1 (items 2-5 fixed in the
  engine, 6 stays the default with the UseSqliteIntegerAffinity opt-in); the non-existent
  DurabilityMode.FlushToDisk is corrected; every item annotated.
- Dead references removed: INDEX.md (4), serialization/README.md (2),
  graphrag/EF_CORE_DOCUMENTATION_INDEX.md (3), analytics/README.md, the issue-56 status table,
  Option.FromNullOrEmpty -> Prelude.Optional, stale v1.7.0 titles, and the never-created
  PERFORMANCE_VALIDATION_AND_ADVICE_CSHARP15_NET11.md pointer.

Guards (DocumentationConsistencyTests, red->green by planting each defect): index paths
resolve; no TOML sample or "Supported Formats: TOML" claim in the server guide; internal
plan-cache APIs not presented as public; every known-issue item annotated; every
install-snippet package id resolves to a project or central package version.

Docs suite 9/0; core suite 2004/0 failed/0 skipped in 89,2 s; build 0 errors.
…oved design, not the code

The Architecture block still presented five rewrite phases (predicate pushdown, subquery
elimination, join reordering) and a "QueryOptimizer.Core" component tree containing
SubqueryOptimizer/JoinReorderer/QueryOptimizerExtensions. It now describes the three steps
Optimize() actually performs (logical view, costing, physical steps), names the flags that are
shape heuristics rather than rewrites, and shows the real component tree including which types
are unwired building blocks.

Docs suite 9/0.
@codecov

codecov Bot commented Oct 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@sonarqubecloud

Copy link
Copy Markdown

@MPCoreDeveloper
MPCoreDeveloper merged commit 204d9f8 into master Oct 11, 2026
14 of 16 checks passed
MPCoreDeveloper added a commit that referenced this pull request Oct 11, 2026
…suites, 3533/0), the corrected landing path (PR #447 already merged; the fix lands as its own PR) and the owner hand-off helper
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant